Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | Go Back
View Post [edit]

Poster: AlecWest Date: Aug 1, 2006 10:35am
Forum: forums Subject: Is there a "24-hour rule" at Archive.org?

Over the last week, I've made 2 attempts to upload a 4.3gb MPEG2 file using SmartFTP. Both times, I was disconnected from the server at the 24-hour mark ... which is somewhat disconcerting since a 4.3gb upload (at your 42-45k/s throttled single-thread upload speed) takes between 28 and 29 hours to complete. And when I attempted to reconnect, the "resume" feature did not work even though SmartFTP was set up to auto-resume.

On the off-chance that it was a SmartFTP problem and not an archive problem, I switched to CoreFTP and am now making my third attempt to upload the file. If it dies at the 24-hour mark, too, then I'll have to assume it's an archive server issue (on at least 3 different servers).

If it IS an archive server issue, a question. How in the heck is someone supposed to upload a normal DVD-sized MPEG2 file to Archive.org servers? On the other hand, if it's a SmartFTP issue, Archive.org might want to reconsider recommending it ... and recommend CoreFTP instead.

Reply to this post
Reply [edit]

Poster: rlmccoy1987 Date: Aug 1, 2006 4:51pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

I am getting the same kind of error. There should be at least 4 or 5 days to upload content. Some content is 4 or 5 gigs big... :(

Reply to this post
Reply [edit]

Poster: AlecWest Date: Aug 1, 2006 4:52pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

rlmccoy1987,

Are you using SmartFTP? That "might" be the problem. It's what I'm testing out now ... using freeware CoreFTP. We'll see what happens 24 hours from now.

Reply to this post
Reply [edit]

Poster: rlmccoy1987 Date: Aug 1, 2006 5:02pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

I use Ipswitch WS_FTP. I have never tried smartftp because it wouldnt work for my xbox's ftp.

Reply to this post
Reply [edit]

Poster: AlecWest Date: Aug 1, 2006 5:13pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

Oh oh ... then it could be a server time-limit after all. I've just replied to an email from Archive Support. We'll see what they have to say.

Reply to this post
Reply [edit]

Poster: rlmccoy1987 Date: Aug 1, 2006 5:16pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

Some people have really slow upload connection. When they try to upload stuff, it times out... I have had this problem many times, I think that it should give us slow people some time to upload, such as a week timelimit...

Reply to this post
Reply [edit]

Poster: AlecWest Date: Aug 1, 2006 5:52pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

rlmccoy1987,

It's not "us." One of the first questions I had was why users are throttled back to 42-45K/s single-thread upload speeds. It's not my ISP. I called to check. Heck, I upload routinely to a German server at 178K/s ... and most domestic servers at ten times the speed allowed by the Archive.

BTW, an Archive support person told me to try again ... so I logged back onto the original server where my previous files existed and found I'd been throttled back to 22K/s. In short, uploading the MPEG2 within 48 hours at that speed would be impossible.

So, I just deleted the whole darn thing. Frankly, I'm sick and tired of being treated like some jerk or neophyte that just fell off the haywagon. And I think it's a sad testament to those who "really want" to upload large files to the Archive - dedicating their time and their computer time to a task where upload speeds are throttled and designated "cut-offs" occur.

Reply to this post
Reply [edit]

Poster: AlecWest Date: Aug 5, 2006 6:20pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

For some reason, some uploaders (even with high-speed broadband like me) find their upload speeds throttled back to 42-45K/s. Persons wanting to upload a 4.3gb DVD-size MPEG2 file would require approximately 28h/45m to upload the entire file. The problem?

There is apparently a designated server cutoff at exactly the 24-hour mark. In short, at the 24th hour of a 28-hour upload, Archive.org servers will simply disconnect you. And when you reconnect to the servers, the "resume" feature will not work (sigh) - even if your FTP program is set up to resume files (like mine was).

Offhand, the simple solution to this problem is to split the MPEG2 file into two halves - uploading each half separately during a separate session. That's what I recently did with this upload:

http://www.archive.org/details/If__I__Had__A__Million

However, when I uploaded the biggest half of the MPEG2 file, the server disconnected me for no apparent reason. But, when I reconnected, I _was_ allowed to resume the upload where I left off.

My first suggestion to the Archive was to bump up the 24-hour cutoff to 48 hours - since we're all given a 48-hour window to complete all uploads anyway. That would solve the issue. But if such a change is impossible, there is another way of beating the 24-hour cutoff rule:

1) Upload half of the file, then manually disconnect.

2) In a new FTP session, reconnect - and you'll be allowed to resume the upload where you left off (assuming your FTP "file exists" parameters are set for "resume").

I'd have done this myself if I'd known earlier that it worked. But when I hit this 24-hour stone wall and couldn't resume afterwards, I naturally assumed that the servers wouldn't allow any resume at all. Not so (grin).

In any case, I sent an email to Archive support asking them to either bump up the 24-hour cutoff to 48 hours - or placing a note on the "create" page:

http://www.archive.org/create.php?identifier=

telling people of this method to "get around" the cutoff if they have a single file that will take longer than 24 hours to upload.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Aug 31, 2006 3:16pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

i don't know of any 24hour limit.

we *do* definitely have a 48 hour limit for items
being checked out.

perhaps 2-3 weeks ago, i updated our "auto checkin" process
to make sure to *not* remove partial uploads if the upload
gets cutoff at the "auto checkin" time.

if you have larger files that you think will exceed the 48 hour limit, we could make some exceptions for those who
let us know.

feel free to reply with observations.

-tracey, webmaster and storage engineer



Attachment: cloudtile.jpg

Reply to this post
Reply [edit]

Poster: AlecWest Date: Aug 31, 2006 4:01pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

All I know is that if I start a 4.3 gig upload at 7:10 PM on Monday, the connection dies promptly at 7:10 PM Tuesday - and that when I log back on, resume won't work.

However, by experiment, I found that I could begin the same upload at 7:10 PM on Monday ... self-terminate the connection at sometime before 7:10 PM on Tuesday ... and when I log back on, resume WILL work - and the upload continues until completed.

I have an always-on broadband connection that has no time limit associated with it. And, my FTP utility has no built in time limit either for single-session uploads. So, if I rule out the ISP and the FTP software, only one culprit is left ... and it's not on my end.

P.S. I'm not the only person who has experienced this. I've talked to 2 others in this forum who experienced the same thing ... using different FTP software and different ISPs in different parts of the country.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffBrak Date: Aug 31, 2006 4:17pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?


Hi,

So this is a cool problem. We do not have any 24 hour ftpd timeouts configured and the master ftpd has been up for weeks on most boxes.

I was going to test this internally, but soon realized that running at 100Mbps for 24 hours meant uploading a 1TB file! Hehe... so I'll test this from home.

Couple of other random theories:
1. ISP's kill a connection after 24 hours of being pegged automatically. This seems unlikely since you mentioned that it's happening to different folks on different ISP's.

2. ISP changes your IP address, causing the connection to go away.

3. Are you using a firewall/cheeseball router thingy. Those have been known to do really stupid things... like time out ssh sessions after 5 minutes. Maybe they do something silly and reset all tcp sessions every 24 hours.

4. OUr own firewall is doing silly things like #3.

I think #3 and #4 are the most likely causes for badness. Home firewalls are notoriously crappy (replacing their firmware with Linux solves most of their problems). They don't have much ram, and dropping connections for the conntrack table is a cheap, if lame, way of not getting support phone calls.

If your box is secure, try uploading while outside of the Linksys.

More in at least 24 hours, I'll start a few uploads tonight and over the weekend.... now if only I had worthy content!

-Joerg

Reply to this post
Reply [edit]

Poster: AlecWest Date: Aug 31, 2006 11:10pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

BRAK WROTE - "I was going to test this internally, but soon realized that running at 100Mbps for 24 hours meant uploading a 1TB file! Hehe... so I'll test this from home."

This may a related issue. My FTP uploads to the archive are throttled back to 42-45k/s - which is why it takes roughly 28-29 hours to upload a 4.3 gig file. My normal upload speeds to other servers are about 10 times that - and one German server I connect to often allows me roughly 175-180k/s uploads.

During a spate of recent upload attempts, I called my ISP (Comcast) to ask if they were throttling back my speed. The tech support rep said no - that it was likely on the server end. Of course, he might have been lying. It could be that "Archive.org" (known for uploads of substantial filesizes) might be on Comcast's s**tlist of connections to throttle.

FWIW, others have also noticed this apparent throttling. But (grin), ISPs could share a common s**tlist.

BRAK WROTE - "1. ISP's kill a connection after 24 hours of being pegged automatically. This seems unlikely since you mentioned that it's happening to different folks on different ISP's."

I'm on Comcast in Portland, Oregon. The other two I communicated with are an Adelphia user in PA and a Verizon user in MD. But again, this could be part of a s**tlist scenario used by more than one ISP. I have a friend in Illinois with his own T3 line and has no throttling issue. And at his speeds, 24-hour timeouts are never reached.

BRAK WROTE - "2. ISP changes your IP address, causing the connection to go away."

My IP is static.

BRAK WROTE - "3. Are you using a firewall/cheeseball router thingy. Those have been known to do really stupid things... like time out ssh sessions after 5 minutes. Maybe they do something silly and reset all tcp sessions every 24 hours."

I never considered that possibility. I use ZoneAlarm but am unaware of any timeout issues with the software.

BRAK WROTE - "4. OUr own firewall is doing silly things like #3."

Could that be the culprit, then?

BRAK WROTE - "If your box is secure, try uploading while outside of the Linksys."

Hmmm ... I use a D-Link router. Next time I upload, I'll disconnect it and run straight off my cable connection to see what happens.

BRAK WROTE - "More in at least 24 hours, I'll start a few uploads tonight and over the weekend.... now if only I had worthy content!"

Thanks ... and I know what you mean. I was certain my last two uploads were public domain ... but one showed up on a superlist and the other showed up on LOC "only" under an abbreviated version of its title (sigh).

I've been tempted to upload the 58-minute "Superman and the Mole-Men." This was the pilot movie to the "Adventures of Superman" TV series in the 50s starring George Reeves. However, unlike the series episodes which were copyrighted by an existing company, the pilot was copyrighted by "Lippert Productions" which is no longer in business. I'm wondering if maybe this might slip through the copyright crack.

Similarly, there were 3 "rocket man" serials from Republic Pictures. Two of them show up on the superlist. But one of them, currently online at the archive (RADAR MEN FROM THE MOON), does not show up on a superlist or LOC ... at least, not under that title. Republic changed the 12-chapter serial into a 12-episode TV series titled COMMANDO CODY - SKY MARSHALL OF THE UNIVERSE - which "is" listed at LOC as being copyrighted. Obviously, RADAR MEN must have slipped through the crack into the public domain by virtue of name change. Likewise, SUPERMAN AND THE MOLE MEN was split into two half-hour episodes in season one of "Adventures" - and retitled "The Unknown People," parts 1 & 2. Could the same "crack" apply to MOLE MEN as well???

But, it's a real bummer to upload something and have it come down. I'm becoming real experienced at that (grin) since I also uploaded the 2 other rocket man serials last year - only to see them come down.

Reply to this post
Reply [edit]

Poster: MatCarpenter Date: Dec 11, 2006 7:49pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

This from Dosdragon.com

[quote]
November 17th 2006 Comcast.net It has come to my attention that Comcast.net has been throttling users by intermittently shutting off a PC's access to the internet. This throttling is intentional, it does not slow the connection but actually HALT or shut off the connection for up to five minutes at a time. I believe they are doing this to prevent flooding, bittorent, down-loaders or VPN access. Though Comcast's terms of service prohibit the use of a VPN on a non-commercial account, their actions are hurting innocent customers. The intentional shut off a user's account can also effect Internet Telephones such as Vontage. These actions could disconnect an important call like 911. After studying the problem I found Comcast is storing MAC address's from personal PCs. This is a gross invasion of privacy and can land the cable company in court. I have provided software for users with a router that allows spoofing the MAC address and restores the connection. The software can be run manually or automatic to detect the loss of internet connection then reset your Network interface card with a new Spoofed MAC address restoring your connection within seconds. You may download the software here http://www.dosdragon.com/antithrottle.zip
[/quote]

http://www.dosdragon.com/antithrottle.zip

Reply to this post
Reply [edit]

Poster: Baire Date: Jun 1, 2007 11:06am
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

Mat, your link is broken or comcast is not allowing me to visit! I'm looking for your program PLEASE HELP! If this is true this could be the begining of a class action?

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Feb 23, 2008 4:39pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

you know, i can also say that it's clear comcast is absolutely doing something.

being IA webmaster, i can directly upload via ssh/scp or rsync to our servers files in the 100s+ MB range.

what i see is a very fast upload for about the first 10-20 seconds, and then it plummets. EVERY time. interestingly, if i want to make my upload go faster, when i watch the upload bitrate plummet, i can "pause" the upload, wait about 30+ seconds, then resume. and i see the exact same "FAST" for a bit, then plummets. so it's clearly comcast (my provider).

i suspect we are all paying for X b/sec uploads, but if we all actually used it, they'd run out of headroom. thus, they put in throttling.

--tracey


Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffDiana Hamilton Date: Feb 23, 2008 7:09pm
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

Latest:
http://www.news.com/Comcast-vs.-BitTorrent-to-be-focus-of-FCC-hearing/2100-1028_3-6231737.html

Comcast vs. ftp has not been directly in the news so much or in governamental sights, but it makes sense that they might throttle multiple protocols if they were messing with one.

This post was modified by Diana Hamilton on 2008-02-24 03:09:15

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Sep 19, 2006 8:55am
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

did your investigations come up with anything?
any successes?

BTW, which server were you uploading to?
something like "XXX-uploads.archive.org"
or something like "iaXXXXXX.us.archive.org"?

--tracey, webmaster

Reply to this post
Reply [edit]

Poster: AlecWest Date: Sep 23, 2006 7:03am
Forum: forums Subject: Re: Is there a '24-hour rule' at Archive.org?

I'm unsure what "investigations" you're referring to. Suffice it to say that I, and others, have uploaded files to the archives at "iaXXXXXX-type" servers and encountered the problem. Exactly (and I mean "exactly") at the 24 hour mark of uploads, the server cuts the connection. And, upon reconnection, resume is not allowed. And, we have different ISPs and even use different FTP utilities. So, the problem isn't ISP related or even FTP related.

The only way "around" this cut-off is to upload a portion of a large file, manually interrupt the upload prior to hitting the 24-hour "brick wall," then reconnect. That "trick" apparently restarts the cutoff clock and allows for the entire file upload to resume and complete.

Maybe...

At this very moment, I'm attempting to upload an MPEG2 file that, under normal constraints. could be uploaded in less than 15 hours. The server I'm attempting to upload it to is "ia331310.us.archive.org." The problem? It allows me to upload for a few seconds, then slows down, then freezes. When I restart my computer (no kidding) and restart SmartFTP, the process repeats - uploading for a few seconds, then slowing down, then freezing up altogether. And BTW, it isn't my ISP. My browser continues to function just fine (like it is now as I post this message). But my FTP connection to that server totally bites.

There are times when I wonder how many people sincerely want to donate uploaded files to the Archive ... are confronted with maddening problems like this ... and just say, "To heck with it." And I wonder how many files the Archive is deprived of because of this.

Question (and I've asked this before at least twice). Would the Archive be willing to consider upping the 48-hour window for uploads to, say, 96 hours??? Not that this would solve the problem I'm having right now. But, assuming I'm just encountering a glitch (as opposed to this being "normal"), 96 hours would give sincere uploaders much needed breathing space. Many of us (like me) have regular jobs and can't sit in front of their computers for double-digit hours to make sure their uploads complete. And many of us would probably like to turn their computers off when they're at work.

This post was modified by AlecWest on 2006-09-23 14:03:28